Design and implementation of a system for automated monitoring of adherence to evidenced-based clinical guideline recommendations (preprint)

Background Clinical practice guidelines are systematically developed statements intended to optimize patient care. However, a gap-less implementation of guideline recommendations requires health care personnel not only to be aware of the recommendations and to support their content, but also to recognize every situation in which they are applicable. To not miss situations in which guideline recommendations should be applied, computerized clinical decision support could be given through a system that allows an automated monitoring of adherence to clinical guideline recommendation in individual patients. Objectives (1) To derive the requirements for a system that allows to monitor the adherence to evidence-based clinical guideline recommendations in individual patients, and based on these requirements, (2) to implement a software prototype that integrates clinical guideline recommendations with individual patient data and (3) to demonstrate the prototype’s utility on a COVID-19 intensive care treatment recommendation. Methods We performed a work process analysis with experienced intensive care clinicians to develop a conceptual model of how to support guideline adherence monitoring in clinical routine and identified which steps in the model could be supported electronically. We then identified the core requirements of a software system for supporting recommendation adherence monitoring in a consensus-based requirements analysis within loosely structured focus group work of key stakeholders (clinicians, guideline developers, health data engineers, software developers). Based on these requirements, we implemented a prototype and demonstrated its functionality by integrating clinical data with a treatment recommendation. Results Based on our conceptual flow chart model of recommendation adherence monitoring in clinical routine, we identified four main requirements of a software system for automated support of recommendation adherence monitoring of in-hospital patients: (i) Ability to interpret guideline recommendations’ semantics and logics, (ii) integration of clinical routine data from various underlying data structures, (iii) automatic adoption of new or updated guideline recommendations, and (iv) user interfaces optimized for distinct groups of users. Using a prototype implementation that fulfills these requirements, we demonstrate how such a system could be applied to monitor guideline recommendation adherence over time in clinical patients. Conclusions The four main requirements identified through our model-based analysis represent the most important aspects that need to be considered when developing a clinical decision support system for monitoring the adherence to evidence-based clinical guideline recommendations in individual patients. As each of the requirements corresponds to a different expertise (guideline development, health data engineering, software development, patient treatment), a modularized software architecture separated by area of required expertise seems favorable. Our prototype successfully demonstrates how such a modular architecture can be implemented to allow real-time monitoring of guideline recommendation adherence. This prototype, which we released as open source to invigorate collaboration, could serve as a basis for further development to integrate guideline recommendations with clinical information systems.

Representation of evidence-based clinical practice guideline recommendations on FHIR (preprint)

BackgroundVarious formalisms have been developed to represent clinical practice guideline recommendations in a computer-interpretable way. However, none of the existing formalisms leverage the structured and computable information that emerge from the evidence-based guideline development process. Thus, we here propose a FHIR-based guideline representation format that is structurally aligned to the knowledge artifacts emerging during the process of evidence-based guideline development. MethodsWe identified the information required to represent evidence-based clinical practice guideline recommendations and reviewed the knowledge artifacts emerging during the evidence-based guideline development process. Then we conducted a consensus-based design process with domain experts to develop an information model for guideline recommendation representation that is structurally aligned to the evidence-based guideline recommendation development process and a corresponding representation based on evidence-based medicine (EBM)-on-FHIR resources. ResultsThe information model of clinical practice guideline recommendations and its EBMonFHIR-based representation contain the clinical contents of individual guideline recommendations, a set of metadata for the recommendations, the ratings for the recommendations (e.g., strength of recommendation, certainty of overall evidence), the ratings of certainty of evidence for individual outcomes (e.g., risk of bias) and links to the underlying evidence (systematic reviews based on primary studies). We created profiles and an implementation guide for all FHIR resources required to represent a complete clinical practice guideline and used the profiles to implement an exemplary clinical guideline recommendation. ConclusionsOur EBMonFHIR-based representation of clinical practice guideline recommendations allows to directly link the evidence assessment process through systematic reviews and evidence grading, and the underlying evidence from primary studies to the resulting guideline recommendations. This not only allows to evaluate the evidence on which recommendations are based on transparently and critically, but also allows for a more direct and in future automatable way to generate computer-interpretable guideline recommendations based on computable evidence.

International Comparisons of Harmonized Laboratory Value Trajectories to Predict Severe COVID-19: Leveraging the 4CE Collaborative Across 342 Hospitals and 6 Countries: A Retrospective Cohort Study (preprint)

Objectives: To perform an international comparison of the trajectory of laboratory values among hospitalized patients with COVID-19 who develop severe disease and identify optimal timing of laboratory value collection to predict severity across hospitals and regions. Design: Retrospective cohort study. Setting: The Consortium for Clinical Characterization of COVID-19 by EHR (4CE), an international multi-site data-sharing collaborative of 342 hospitals in the US and in Europe. Participants: Patients hospitalized with COVID-19, admitted before or after PCR-confirmed result for SARS-CoV-2. Primary and secondary outcome measures: Patients were categorized as ''ever-severe'' or ''never-severe'' using the validated 4CE severity criteria. Eighteen laboratory tests associated with poor COVID-19-related outcomes were evaluated for predictive accuracy by area under the curve (AUC), compared between the severity categories. Subgroup analysis was performed to validate a subset of laboratory values as predictive of severity against a published algorithm. A subset of laboratory values (CRP, albumin, LDH, neutrophil count, D-dimer, and procalcitonin) was compared between North American and European sites for severity prediction. Results: Of 36,447 patients with COVID-19, 19,953 (43.7%) were categorized as ever-severe. Most patients (78.7%) were 50 years of age or older and male (60.5%). Longitudinal trajectories of CRP, albumin, LDH, neutrophil count, D-dimer, and procalcitonin showed association with disease severity. Significant differences of laboratory values at admission were found between the two groups. With the exception of D-dimer, predictive discrimination of laboratory values did not improve after admission. Sub-group analysis using age, D-dimer, CRP, and lymphocyte count as predictive of severity at admission showed similar discrimination to a published algorithm (AUC=0.88 and 0.91, respectively). Both models deteriorated in predictive accuracy as the disease progressed. On average, no difference in severity prediction was found between North American and European sites. Conclusions: Laboratory test values at admission can be used to predict severity in patients with COVID-19. There is a need for prediction models that will perform well over the course of the disease in hospitalized patients.

Validation of a Derived International Patient Severity Phenotype to Support COVID-19 Analytics from Electronic Health Record Data (preprint)

Introduction. The Consortium for Clinical Characterization of COVID-19 by EHR (4CE) includes hundreds of hospitals internationally using a federated computational approach to COVID-19 research using the EHR. Objective. We sought to develop and validate a standard definition of COVID-19 severity from readily accessible EHR data across the Consortium. Methods. We developed an EHR-based severity algorithm and validated it on patient hospitalization data from 12 4CE clinical sites against the outcomes of ICU admission and/or death. We also used a machine learning approach to compare selected predictors of severity to the 4CE algorithm at one site. Results. The 4CE severity algorithm performed with pooled sensitivity of 0.73 and specificity 0.83 for the combined outcome of ICU admission and/or death. The sensitivity of single code categories for acuity were unacceptably inaccurate - varying by up to 0.65 across sites. A multivariate machine learning approach identified codes resulting in mean AUC 0.956 (95% CI: 0.952, 0.959) compared to 0.903 (95% CI: 0.886, 0.921) using expert-derived codes. Billing codes were poor proxies of ICU admission, with 49% precision and recall compared against chart review at one partner institution. Discussion. We developed a proxy measure of severity that proved resilient to coding variability internationally by using a set of 6 code classes. In contrast, machine-learning approaches may tend to overfit hospital-specific orders. Manual chart review revealed discrepancies even in the gold standard outcomes, possibly due to pandemic conditions. Conclusion. We developed an EHR-based algorithm for COVID-19 severity and validated it at 12 international sites.

International Electronic Health Record-Derived COVID-19 Clinical Course Profile: The 4CE Consortium (preprint)

We leveraged the largely untapped resource of electronic health record data to address critical clinical and epidemiological questions about Coronavirus Disease 2019 (COVID-19). To do this, we formed an international consortium (4CE) of 96 hospitals across 5 countries (www.covidclinical.net). Contributors utilized the Informatics for Integrating Biology and the Bedside (i2b2) or Observational Medical Outcomes Partnership (OMOP) platforms to map to a common data model. The group focused on comorbidities and temporal changes in key laboratory test values. Harmonized data were analyzed locally and converted to a shared aggregate form for rapid analysis and visualization of regional differences and global commonalities. Data covered 27,584 COVID-19 cases with 187,802 laboratory tests. Case counts and laboratory trajectories were concordant with existing literature. Laboratory tests at the time of diagnosis showed hospital-level differences equivalent to country-level variation across the consortium partners. Despite the limitations of decentralized data generation, we established a framework to capture the trajectory of COVID-19 disease in patients and their response to interventions.